File systems store files and store information about files. The information stored in files may be referred to as data. The information about files may be referred to as metadata. The metadata may include, for example, a file name, a file size, a file parent, a file descendant, a file access time, a file owner, file permissions, and other information. Some of the metadata for an individual file may be stored in a data structure known as an inode. The inodes and metadata for a file system may also be stored collectively. The metadata has both structure and content. For example, the content (e.g., metadata) may be stored in a set of data structures (e.g., btrees). When the metadata is stored in one or more trees (e.g., btrees), there may be a root inode associated with the top of the set of trees. When the data in a file or about a file changes, a file system may want to update the metadata about that file. For example, if the contents of a file are changed, the file system may want to memorialize the time at which the change was made and by whom the change was made.
A file system organizes and makes files available in, for example, the familiar hierarchical directory, sub-directory, file structure. A shared disk file system may run on hosts that are connected to the same disk array on a storage area network (SAN). A shared disk file system may hierarchically store large data sets on disk, on tape, and in other locations. The tape may be on-line in a data center and off-line in a vault. Having both on-line and off-line tape facilitates economizing on the costs of storage while allowing for large scale data retention and archiving, all while protecting against hardware failures.
To protect the metadata concerning the files organized and made available by a file system, the file system may include a metadata dump facility that contains a mirror of the file system metadata. The metadata dump may be referred to as a metadump. The file system may seek to keep the metadump updated in real time as the metadata changes in the file system. The metadump may be intended to facilitate, for example, accelerated disaster recovery. In the event of a disaster, the metadump can be used to restore metadata in a wholesale manner for the file system it mirrors. Restoring the metadata may include rebuilding the structure(s) in which the metadata was stored and then populating those structure(s).
Keeping the metadump up-to-date may be a significant challenge due to the realities of large file systems. For example, making a change to a file may require the file system to perform updates to several independently stored pieces of metadata. But the underlying storage may not support making the several updates as an atomic operation. This set of updates may take the file system from one consistent state to another. Undesirable conditions may arise if a series of update operations are only partially recorded. Thus, a file system may be required to treat a series of operations as a transaction. Example transactions may include allocating space for a file, creating a file, updating a file, deleting a file, or other operations. With multiple hosts and multiple users, there may be a significant number of transactions.
As a file system became large, it may have been difficult, if even possible at all in a relevant time frame, to scan through the file system to find files to create and then maintain a metadump. Additionally, the metadump may have been stored as a file whose format mirrored the primary metadata format. This may have made it difficult, if even possible at all in a relevant time frame, to scan through all the files represented in the metadump file during a restore operation. Since the metadump is supposed to be an accurate, up-to-date in real time mirror of the file system metadata, creating a metadump may have required taking the file system off-line and making the file system unavailable while the file system was scanned and the metadump created so that no changes would be made and missed during creation. Similarly, using a metadump to restore a file system may have required keeping the file system off line and unavailable while the restore occurred. Having a file system off-line for any time, and particularly for an extended time (e.g., hours, days) is inconvenient at best, and generally unacceptable.
The challenges associated with keeping a metadump up-to-date in real-time also include the difference in latency between memory and non-memory (e.g., disk, tape) storage. This latency can produce conditions where changes made in one area (e.g., memory) are out of sync with changes made in another area (e.g., disk). Additionally, this latency motivates a file system to store in memory changes that are to be made to data on disk and then to make the actual changes on disk at a later time. For example, a series of reads and writes to a file may be made virtually in memory and then only made physically on disk at a later time. While this delayed update approach may solve one problem associated with excessive random input/output (i/o), it may produce another problem associated with memory and disk being out of sync. The file system metadata may indicate that a change has been made, and that change may have been performed in memory, but the actual underlying data on disk may not have been changed.
For all these reasons, improvements to creating, maintaining, and using file system metadumps are sought.